On Modeling CPU Utilization of MapReduce Applications

In this paper, we present an approach to predict the total CPU utilization in terms of CPU clock tick of applications when running on MapReduce framework. Our approach has two key phases: profiling and modeling. In the profiling phase, an application is run several times with different sets of MapReduce configuration parameters to profile total CPU clock tick of the application on a given platform. In the modeling phase, multi linear regression is used to map the sets of MapReduce configuration parameters (number of Mappers, number of Reducers, size of File System (HDFS) and the size of input file) to total CPU clock ticks of the application. This derived model can be used for predicting total CPU requirements of the same application when using MapReduce framework on the same platform. Our approach aims to eliminate error-prone manual processes and presents a fully automated solution. Three standard applications (WordCount, Exim Mainlog parsing and Terasort) are used to evaluate our modeling technique on pseudo-distributed MapReduce platforms. Results show that our automated model generation procedure can effectively characterize total CPU clock tick of these applications with average prediction error of 3.5%, 4.05% and 2.75%, respectively. KeywordCPU utilization, CPU clock tick, MapReduce, Modeling, Prediction, Regression